Cleansing Data for Mining and Warehousing

Given the rapid growth of data it is important to extract mine and discover useful information from databases and data ware houses The process of data cleansing is crucial because of the garbage in garbage out principle Dirty data les are prevalent because of incorrect or missing data values inconsistent value naming conventions and incomplete information Hence we may have multiple records refer ing to the same real world entity In this paper we examine the problem of detecting and removing duplicating records We present several e cient techniques to pre process the records before sorting them so that potentially matching records will be brought to a close neighbourhood Based on these techniques we implement a data cleansing system which can detect and remove more duplicate records than existing methods